John Li's Blog Site

Gitlab CI Shell Executor Experience vs. Docker Executor

April 19, 2019

Gitlab is becoming a more one stop shop for software engineers. The first impression is it is very simple like some of the introduction videos of course. :-)

As you can see in this brief introduction to GitLab CI

And here is a GitLab CI Quick Start

I recently tried to leverage the Gitlab CI for some of my projects and I am really impressed with the gitlab way of integration and simplicity. But the meanwhile you need to be prepared for the massive information there. You can easily get lost if you don’t really have a deep understanding of how things work there.

For me the most important reference page is GitLab CI/CD Pipeline Configuration Reference

Here is the doc for GitLab Runner Executors

Most of the GitLab CI projects are using docker executor but I eventually gave up on it for one of my project because of the customization I need. My app is Gatsby based and it has quite some files in node_modules. This type of project has a GitLab CI template as below:

# This file is a template, and might need editing before it works on your project.
image: node:latest

# This folder is cached between builds
# http://docs.gitlab.com/ce/ci/yaml/README.html#cache
cache:
  paths:
  - node_modules/

pages:
  script:
  - yarn install
  - ./node_modules/.bin/gatsby build --prefix-paths
  artifacts:
    paths:
    - public
  only:
  - master

Here is a typical runner log:

Running with gitlab-runner 11.10.0 (3001a600)
  on test-runner FYmx5LMB
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...
Using docker image sha256:9289251188dedce7f1ddbb3edbd4473bec2e0f7ba25713c84e9da2625b95621a for node:latest ...
Running on runner-FYmx5LMB-project-11928875-concurrent-0 via e20241b601c7...
Reinitialized existing Git repository in /builds/jli/gdemo/.git/
Fetching changes...
From https://gitlab.com/jli/gdemo
   a1f1d6f..eedc449  master     -> origin/master
Checking out eedc449b as master...
Removing .cache/
Removing node_modules/
Removing public/

Skipping Git submodules setup
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
$ echo "Before script section"
Before script section
$ yarn install
yarn install v1.15.2
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@1.2.7: The platform "linux" is incompatible with this module.
info "fsevents@1.2.7" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "gatsby > eslint-plugin-graphql@2.1.1" has incorrect peer dependency "graphql@^0.12.0 || ^0.13.0".
warning "gatsby > express-graphql@0.6.12" has incorrect peer dependency "graphql@^0.10.0 || ^0.11.0 || ^0.12.0 || ^0.13.0".
warning "gatsby > graphql-tools@3.1.1" has incorrect peer dependency "graphql@^0.13.0".
[4/4] Building fresh packages...
Done in 75.85s.
$ ./node_modules/.bin/gatsby clean
info Deleting .cache, public
info Successfully deleted directories
$ ./node_modules/.bin/gatsby build --prefix-paths
success open and validate gatsby-configs — 0.008 s
success load plugins — 0.403 s
success onPreInit — 0.007 s
success delete html and css files from previous builds — 0.007 s
success initialize cache — 0.007 s
success copy gatsby files — 0.012 s
success onPreBootstrap — 0.009 s
success source and transform nodes — 0.048 s
success building schema — 0.194 s
success createPages — 0.000 s
success createPagesStatefully — 0.032 s
success onPreExtractQueries — 0.002 s
success update schema — 0.023 s
success extract queries from components — 0.100 s
success run graphql queries — 0.185 s — 7/7 38.16 queries/second
success write out page data — 0.003 s
success write out redirect data — 0.000 s

success Build manifest and related icons — 0.234 s
success onPostBootstrap — 0.236 s

info bootstrap finished - 3.52041038 s

success Building production JavaScript and CSS bundles — 7.506 s
success Building static HTML for pages — 0.685 s — 4/4 19.47 pages/second
info Done building in 11.715142753 sec

And the other stages in the pipeline also have the similiar entries in the beginning of the logs

Running with gitlab-runner 11.10.0 (3001a600)
  on test-runner FYmx5LMB
Using Shell executor...
Running on e20241b601c7...
Reinitialized existing Git repository in /home/gitlab-runner/builds/FYmx5LMB/0/jli/gdemo/.git/
Fetching changes...
Checking out 51c50abe as master...
Removing .cache/
Removing node_modules/
Removing public/
Skipping object checkout, Git LFS is not installed.

Skipping Git submodules setup
Checking cache for default...
Runtime platform                                    arch=amd64 os=linux pid=3846 revision=3001a600 version=11.10.0
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Downloading artifacts for build1 (199708756)...
Runtime platform                                    arch=amd64 os=linux pid=3870 revision=3001a600 version=11.10.0
Downloading artifacts from coordinator... ok        id=199708756 responseStatus=200 OK token=xdF9zTdC
$ echo "Before script section"

You can tell from the logs that the node_modules was cached and restored to avoid repeatitive module installation, this can improve the pipeline performance.

In the deploy part of the pipeline, my project needs to use git clone the other project and merge the generated content into that project, then commit and push the changes back to the other project’s repository. At this moment I was using shell executor to implement the logic since I need to put in the ssh credential for that project’s read/write access, and I don’t want to clone that project everytime the pipeline runs. The Shell Executor works but it needs to have some setup before pipeline works.

Maybe in the future I can figure out a way to use the Docker executor while at the same time to use the ssh credential and avoid the clone everytime. Please share your experience if you have a better way. Thanks for reading.

John Li's stuff for work, study and social activities

You can follow him on Twitter